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Regulatory peer review can accomplish many functions. Any particular review, however, 
must be focused on a limited number of goals, or it will fail to achieve any of them — with 
the possible exception of checking the box for having had the review. 

Regulatory peer review requires a balancing of conflicting desires: 

■ • timely review of the document versus the time and cost for in-depth review, 

• * comprehensive review of the entire document by each member of the review 

committee versus an assigned or segmented review where each specialist focuses 
on his/her part, 

• • review of the document of concern essentially in isolation, i.e., as a self-contained 

document, versus inclusion of any/all other materials, e.g., what else could/should 
be in the document, and 

• • best possible review with unconstrained resources versus sufficiently good/ 

appropriate to achieve policy/legislative goals and constraints. 

These conflicts are not independent. Indeed, as presented below, each can significantly 
affect the other. 

Issue: To what extent is the review solely of the existing document, or are other data 
and documents intended to be included In the review? 

The chronological time for reviewing many regulatory documents is relatively short and, 
therefore, generally does not allow for review of external material without existing 
knowledge on the part of the peer reviewers. This suggests that every topic of concern 
must be identified and that at least one reviewer would be an expert in that field or topic. 

• • If the critical issues have not been identified by those who pick the reviewers, the 

issues may not be identified — much less reviewed. 

• * Those writing the document or picking the panel may not understand — or may not 

wish to address — certain issues important to others. If such factors affect selection 
of reviewers, critical issues may not be (fully) addressed. 

To the extent that a member of the review panel brings additional data into the process, 
these data will be biased. 

• • The reviewer will select those data s/he believes critical and/or useful to make a 

point. 

• • The additional information will therefore depend on the members of the panel and 

their expertise. Issues and expertise not identified may not be given this additional 
consideration. 

• • The other reviewers will not be able to review the new information in the same 

manner as the rest of the document. 
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Issue: If the review is primarily of the document itself, to what extent will the should 
the document be reviewed? 

While the size of regulatory documents generally precludes careful parsing of every 
sentence, sentences or paragraphs may be cited subsequently, and given credibility as 
having been "peer-reviewed." 

Most would agree that every (major) concept and model should be reviewed, but those 
who prepare the document may disagree with both the reviewers and those who must 
comply with the requirements as to which are the critical concepts and models. For 
example, if a mixture of chemicals were being reviewed and EPA’s hazar d index were 
used to evaluate the mixture, are critiques concerning the limitations of the hazard index 
model itself within the purview of review process? Reviewing all issues from first 
principles is not possible, but neither is it reasonable to assert additional acceptance of a 
model by a peer review if the reviewers are asked to assume that "standard regulatory 
practices" are valid for the purposes of the review. 

Issue: Chronological and compensated time may be constrained by factors not under 
the control of those involved with the review process. 

Allowing sufficient time for an independent review of the primary data/documents may not 
be feasible. 

* * Such a review could take months, significantly delaying action. 

Many experts, for various reasons, could not participate in such an extended review 
without more compensation. Increased compensation may compromise (perceived) 
independence of the reviewers. 

* * Without an independent review of the primary data, must the reviewers assume that 

all of the relevant data are presented, properly represented, and accurately 
analyzed? A review of the public comments on many government documents 
suggests that this assumption is not always valid. 

Issue: In addition to technical expertise, what other qualifications should be 
considered in selecting peer reviewers? 

To keep panels at manageable size, peer reviewers who can cover several issues are often 
selected. Other considerations must be identified early in the selection process, else one 
may be attempting to identify a developmental biologist with some expertise with virology 
with other characteristics to achieve the desired balance. 

At the outset, balance must be distinguished from bias. Everyone has biases, at a minimum 
based on personal professional experience with particular substances, processes, or 
models. Balancing known or inferred biases can ensure views are appropriately discussed 
and evaluated. 
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While diversity within a review pane! is essential for reviews that include policy, 
diversity will also often improve technical review. For example, inclusion of younger 
scientists may increase consideration of state-of-the-art options than reliance on those of us 
who have grown comfortable with older techniques. Women and minorities are often 
considered, but geographic diversity may be critical. Knowledge of rural or agricultural 
culture and common practices are lacking in some of us who have only lived in urban or 
suburban locations. 

Issue: To the extent questions or issues are presented to the reviewers, they will bias 
the review. 

Identifying issues and models as critical will focus the review on those topics. Given 
limited time and other resources, the review may exclusively address those topics — even 
if one of the questions is the ubiquitous invitation to address other issues. To the extent 
reviewers are asked to focus on particular issues, chronological time alone may prevent 
their ability to reflect creatively on other issues. 

The type of question asked can also bias the review. For example, it is quite different to 
ask if all of the data were collected and reviewed than to consider whether the data were 
appropriately represented (which may require review of the primary documents) or 
whether the data as a whole were analyzed to an appropriate conclusion. 


If the reviewers have a finite meeting time during which to address specific questions, this 
constraint may also prevent discussion of additional topics. Facilitators who are not 
knowledgeable about the nuances of the issues may also constrain consideration of 
additional topics. 

Issue: What level of sufficiency is required for the document to pass muster? 

Several levels of sufficiency can be deemed appropriate, for example: 

• • the best possible, 

• • representative of the field, 

• * correctly portraying the field (by whose criteria?), 

• * correctly/adequately analyzing the data/information presented, 

• • correctly/adequately interpreting the data/information presented, 

• • one of several acceptable outcomes (should/must others also be presented?), or 

• * sufficiently acceptable, i.e., contains no obvious errors. 

For example, one could perform a worst-case model of tides based on the Bay of Fundy. 
Issues for peer review of the analysis could include the following issues. 

• ♦ Are the correct data for case analyzed? 

• * Is the analysis representative? 

• * Is the model accurate for other situations? 
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• • Should the reviewers be expected to check other data and/or other models to 
determine to what extent this analysis is representative? 

Some Selected Solutions 

> The peer review panel as well as the final document should clearly state the goals 
and functions of the peer review for that document. 

> Issues, models, and concepts specifically identified for peer review should be 
noted. 

> The document should include a caveat that states that, while the concepts and 
models were peer reviewed, no individual sentence or paragraph has necessarily 
been given that level of consideration. To the extent desired, the panel may be 
asked to focus on particular sections, e.g., the Executive Summary, or specific 
conclusions, e.g., bullet summaries within the text. If specifically reviewed, these 
could then be designated by format or font as peer-reviewed text. 
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